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Abstract 

Background: Quality indicators (Qls) are used in many liealtlicare settings to measure, compare, and improve 
quality of care. For the efficient development of high-quality Qls, rigorous, approved, and evidence-based 
development methods are needed. Clinical practice guidelines are a suitable source to derive Qls from, but no 
gold standard for guideline-based Ql development exists. This review aims to identify, describe, and compare 
methodological approaches to guideline-based Ql development. 

Methods: We systematically searched medical literature databases (Medline, EMBASE, and CINAHL) and grey 
literature. Two researchers selected publications reporting methodological approaches to guideline-based Ql 
development. In order to describe and compare methodological approaches used in these publications, we 
extracted detailed information on common steps of guideline-based Ql development (topic selection, guideline 
selection, extraction of recommendations, Ql selection, practice test, and implementation) to predesigned 
extraction tables. 

Results: From 8,697 hits in the database search and several grey literature documents, we selected 48 relevant 
references. The studies were of heterogeneous type and quality. We found no randomized controlled trial or other 
studies comparing the ability of different methodological approaches to guideline-based development to generate 
high-quality Qls. The relevant publications featured a wide variety of methodological approaches to guideline- 
based Ql development, especially regarding guideline selection and extraction of recommendations. Qnly a few 
studies reported patient involvement. 

Conclusions: Further research is needed to determine which elements of the methodological approaches 
identified, described, and compared in this review are best suited to constitute a gold standard for guideline-based 
Ql development. For this research, we provide a comprehensive groundwork. 



Background 

According to the definition of the Institute of Medicine 
(1990), quality of care is the "degree to which health 
services for individuals and populations increase the 
likelihood of desired health outcomes and are consistent 
with current professional knowledge" [1,2]. Increasingly, 
quality indicators (Qls) are employed to assess and 
improve the quality of care in many healthcare settings 
[1,3-5]. Qls are measurable items referring to structures, 
processes, and outcomes of care [6]. They imply a judg- 
ment on the quality of care provided. However, the 
interpretation of such performance assessments can 
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have far-reaching consequences, for instance, in applica- 
tion to pay-for-performance models. Hence, the devel- 
opment of Qls should be based on a systematic 
approach that ensures transparency and produces high- 
quality standards [7]. Important attributes of high-qual- 
ity Qls are their relevance to the selected problem and 
field of application, their feasibility, and their reliability. 
They further need to be easily understandable for provi- 
ders and patients, changeable by behavior, achievable, 
and measurable with high validity [8,9]. To ensure con- 
tent and construct validity, Qls need to be evidence 
based and should have a strong correlation with the 
actual quality of care provided, respectively [9,10]. The 
reliability of Qls in regard to their level of measurement 
error can be assessed by an evaluation of the intra- and 
inter-observer reliability [11]. 
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State-of-the-art methodological approaches to QI 
development have been described in several studies 
[12-15], and a large body of literature exists evaluating 
their strengths and limitations [13,16,17]. However, to 
date, no study of which we are aware exists that system- 
atically compares different methodological approaches 
to QI development with respect to their ability to gener- 
ate QIs that improve the quality of the particular health- 
care aspects they were designed for. 

Developing QIs is an expensive and time-consuming 
process. They are usually specific to certain healthcare 
settings and, as a result, cannot always be applied to 
other settings without an adequate adaption process 
[17]. A time-efficient and resource-saving approach is 
either to generate QIs from clinical guidelines already 
available or to couple the process of guideline develop- 
ment with the formulation of appropriate QIs [18,19]. 
Due to the aim of clinical practice guidelines to improve 
quality-of-care processes in practices and care institu- 
tions, guideline-based QIs predominantly relate to pro- 
cess quality. However, no gold standard exists for 
guideline-based QI development [10,20,21]. 

Blozik et al. [20] recently conducted a survey among 
members of the Guideline International Network (G-I-N 
[Guidelines International Network, Perthshire, Scot- 
land]) that shows that even among working groups spe- 
cializing in guideline and QI development, a wide 
variety of methodological approaches are used. A gold 
standard would help to standardize procedures, foster 
transparency, and improve efficiency of resources used. 

This review aims to identify, describe, and compare 
methodological approaches to guideline-based QI devel- 
opment. By pooling the available knowledge and 
appraising strengths and limitations, we intend to pro- 
vide the groundwork necessary for defining a gold stan- 
dard for the development of QIs from clinical practice 
guidelines. To achieve this, we addressed the following 
research questions: 

1. Which methodological approaches to guideline- 
based development of QIs have been described so far? 

2. What are the strengths and limitations of the meth- 
odological approaches described regarding their ability 
to generate high-quality QIs? 

3. Do methodological approaches to the development 
correlate with the quality of QIs they produce? 

Methods 

We carried out a systematic literature search across 
three electronic databases: MEDLINE (US National 
Library of Medicine, Bethesda, MD, USA), the Excerpta 
Medica database (Embase [Elsevier B.V., New York, NY, 
USA]; both via OvidSP® [Ovid Technolo gies. Inc., New 
York, NY, USA]) to cover articles in medical journals 
that are not included in MEDLINE, and the Cumulative 



Index to Nursing and Allied Health Literature (CINAHL 
[EBSCO Publishing, Ipswich, MA, USA]) to include arti- 
cles published in the field of nursing and the allied 
health professions. The query date of all three databases 
was April 22, 2010. The search included literature from 
the earliest records available in the databases up to the 
search date. Duplicates were eliminated both manually 
and automatically. To identify articles for review, we 
linked three search columns using the Boolean operator 
"and": quality indicators, guidelines, and development. 
We combined several search terms with the Boolean 
operator "or" in order to operationalize the search terms 
(the MEDLINE search algorithm can be found in Addi- 
tional file 1: Table SI and was slightly adapted for 
Embase and CINAHL). We drew several search terms 
from the controlled vocabularies used for subject index- 
ing in MEDLINE (i.e.. Medical Subject Headings 
[MeSH]), Embase {i.e., EMTREE), and CINAHL {i.e., 
CINAHL Subject Headings). We searched three data- 
bases for ongoing studies (Current Controlled Trials 
[Springer Science & Business Media, New York, NY, 
USA], HSRProj [Health Services Research Projects in 
Progress, US National Library of Medicine, Bethesda, 
MD, USA], UKCRN-Portfolio [United Kingdom Clinical 
Research Network, National Institute for Health 
Research, London, UK] [22]). In addition, we screened 
the reference lists of all retrieved publications included 
in the final review. From the relevant literature and the 
G-I-N database, we derived contact information of insti- 
tutions and working groups in the field of guideline and 
QI development. We scanned relevant government and 
institutional websites in order to obtain web-published 
documents such as method papers (for details of web- 
sites searched, see Additional file 2: Table S2). Finally, 
we consulted colleagues with a research interest in QI 
to point out articles not identified during our database, 
websites, and reference list search. 

Two reviewers independently screened all obtained 
references for eligibility in a three-stage screening pro- 
cess. Discrepancies were solved by consensus. Articles 
were considered for inclusion if they reported at least 
one methodological approach to guideline-based QI 
development and if they were published in English, 
French, or German. All study and publication types 
were included. 

The detailed reporting of the individual development 
steps (see next paragraph) in publications describing 
methodological approaches to QI development is indis- 
pensable for their reconstruction-be it for the purpose 
of process evaluation (as we did) or in order to apply 
methodological approaches to QI development in other 
settings. We therefore excluded studies at the full-text 
screening stage that did not describe the extraction of 
recommendations from clinical guidelines in detail, as 
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this was the process of particular interest to this review. 
Details of the selection process, including exclusion cri- 
teria at the abstract-screening stage, are summarized in 
Figure 1. 

Two researchers independently extracted data from 
the relevant literature to a predesigned data extraction 
form (see Additional file 3: Table S3); discrepancies 
were solved by consensus. In order to describe and to 
compare methodological approaches to guideline-based 
QI development, we developed an a priori framework of 
the QI development process. For this purpose, we iden- 
tified six steps that most methodological approaches to 
guideline-based QI development have in common with 
regard to function and succession but that differ in their 
design from one methodological approach to another. 
Through a preliminary search and analysis of a select 
number of key publications, we identified six develop- 
ment steps: (1) topic selection, (2) guideline selection, 
(3) extraction of recommendations, (4) QI selection, (5) 
practice test, and (6) implementation (see Figure 2). The 
data extraction form was specifically designed to include 
(a) information about the methodological approach to 
these six development steps and (b) items necessary to 
perform a quality assessment of the relevant studies. For 
steps 1 to 4, we extracted information about how and 
by whom the specific development step was conducted, 
such as selection criteria for topics, guidelines, and 



Primary hits (database 
search): n = 8,697 




Figure 1 Flowchart summarizing the screening process. 
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Figure 2 Overview of the process of guideline-based QI 
development 



recommendations, as well as participants. The two 
development steps specific to guideline-based QI devel- 
opment (compared to QI development from other 
sources) were investigated in more detail, namely, guide- 
line selection and extraction of recommendations. In 
addition to the above-mentioned selection criteria, we 
collected information about the selected guidelines (Was 
some sort of quality assessment conducted? Were all 
selected guidelines listed in the publication?), as well as 
the extracted recommendations (Were they reported at 
all? If yes, were the source guideline and the underlying 
level of evidence made transparent?). For an overview of 
all selected information on guideline selection and 
extraction of recommendations, see Table 1. 
Due to the wide variety of study and publication types 
and the overlap of the quality assessment and the 
assessment of methodological approaches, we limited 
the quality assessment to items covering funding infor- 
mation, the reporting of study and publication type, and 
the reporting of duration and time frame of the study. 

Following data extraction and identification of the 
methodological approaches to each of the above-listed 
development steps, we focused on analyzing the similari- 
ties and differences among the identified methodological 
approaches. The results are presented following further 
elaboration of the six development steps introduced 
above. We discuss our results in context of the current 
literature in the Discussion section. 

Results 

Search findings and literature selection 

We identified a total of 8,697 potentially relevant arti- 
cles, of which 8,468 were excluded based on their titles 
or abstracts (see Figure 1 for details regarding the 
screening process). No additional articles were identified 
through expert consultation. We conducted full-text 
reviews of the remaining 229 articles and an additional 
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Table 1 Information extracted relating to guideline selection and extraction of recommendations 



Guideline selection 


Extraction of recommendations 


Were Qls developed from 

• one guideline, 

• more than one guideline, or 

• guidelines and other sources? 


Were 

• all recommendations or 

• a selection of recommendations extracted? 


Which criteria for guideline selection were reported? 


If not all recommendations were extracted, which criteria were reported for 
their selection? 


Did the authors report a critical appraisal of selected guidelines? 


Who did extraction recommendations? 


Were the selected guidelines listed in the publication? 


Which criteria were reported for the selection of persons involved in 
recommendation extraction? 


Who selected the guidelines? 


Were the extracted recommendations reported in the publication or additional 
files available to the reader? 


Which criteria were reported for the selection of persons involved 
in guideline selection? 


Did the authors report sources/levels of evidence of the extracted 
recommendations? 



Ql = quality indicator 



eight articles identified in reference lists and in the grey 
literature. The final review included 48 articles. 

Of the 48 articles in the final review, 10 papers described 
methodological approaches to guideline-based QI develop- 
ment in general (referred to as "method papers") 
[1,7,23-30], and 32 articles [31-62] addressed the guideline- 
based QI development for a certain clinical topic (referred 
to as "topic papers"). An additional six papers [10,19,63-66] 
comprised a detailed description of a method as well as its 
application for a certain clinical topic (referred to as 
"method + topic papers"). None of the selected pubhca- 
tions was a controlled study comparing one development 
method to another. All journal articles were published in 
English; two of the method papers published via institu- 
tional websites [25,26] were written in German. 

In not disclosing the funding source and time frame of 
the study and in not explicitly reporting the study type, 
many of the publications did not meet our basic quality- 
assessment criteria (for details, see Table 2). 

The identified relevant studies originate from many 
different institutions and working groups, only a few of 
which have published more than one relevant study on 
guideline-based QI development {e.g., the Dutch IQ 
healthcare [University of Radbound, Nijmegen, The 
Netherlands]). 

Tables 2, 3, and 4 provide an overview of the charac- 
teristics of all included publications. Figure 3 provides a 
comprehensive overview of all methodological 
approaches identified. 

Unless indicated otherwise, numbers of studies 
referred to in the following paragraphs always relate to 
all 48 studies of the final review pool. 

Topic selection 

Criteria for the selection of a clinical topic for QI devel- 
opment were detailed in 33 publications. The most fre- 
quently reported criteria were 



• the public health relevance of a topic (mentioned in 
18 publications), 

• the existence of a gap between potential and actually 
achieved quality of healthcare (mentioned in 16 
publications). 

Other reported criteria were uncertainty about the 
quality of care provided for a specific healthcare setting 
(mentioned in six publications), the economical impact 
of a specific healthcare problem (mentioned in six publi- 
cations), and the individual impact on the quality of life 
(mentioned in four publications). 

Guideline selection 

In 16 studies, Qls were developed from a single guide- 
line, whereas in seven studies more than one guideline 
was used to derive Qls. Twenty studies detailed other 
sources, such as existing QI databases, in addition to 
clinical guidelines. 

Only eight of the authors who developed Qls from 
more than one source provided a transparent descrip- 
tion of the respective sources of final Qls. 

Criteria for the selection of guidelines from which the 
Qls were derived were reported in 10 publications. 
Reported criteria were 

• the methodological quality, 

• the up-to-dateness, 

• the eligibility of a guideline for the selected topic (e. 
g., with regard to the specific setting). 

In 15 publications a critical appraisal of the used 
guidelines was reported based on the Appraisal of 
Guidelines Research and Evaluation in Europe (AGREE) 
instrument [67] or similar quality criteria. 

Whilst participants in guideline selection are often 
mentioned, at least indirectly, for instance by being 
referred to as "the authors", criteria for their selection 
were reported in only four publications. These selection 
criteria were 
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Table 2 Characteristics of included references: General characteristics and quality assessment 



General characteristics 






Quality assessment 




Reference 


Institution 


Topic 


Setting 


Study/publication 
type nf)entioned 


Study 

duration 

mentioned 


Funding 


IVIethod papers 


AZQ (2009) 


AZQ (Berlin, DE) 






No 


n/a 


Unclear 


AHCPR (1995) 


AHRQ (Rockville, MD, US) 


- 


- 


No 


n/a 


Unclear 




AUDpi /'D^r-U;illci ^/l^ 1 K'l 






Yes - report 


n/a 

n/ d 


Combined 

public/ 

private 


AQUA (2010) 


AQUA (Gdttingen, DF) 






Yes - method 
paper 


n/a 


Unclear 


Baker and Fraser 
(1995) 


Eli Lilly National Clinical Audit 
Centre (Leicester, UK) 






Yes - review 


n/a 


Unclear 


Bergman (1999) 


Dept. of Pediatrics, Stanford School 
of Medicine (Palo Alto, CA, US) 






No 


n/a 


Unclear 


Califf et al. (2002) 


DCRI (Durham, NC, US) 






Yes - state-of-the- 
art paper 


n/a 


Public 


Campbell et ai. 
(2002) 


NPCRDC (Manchester, UK) 






Yes - review 


n/a 


Unclear 


Graham ef al. 
(2009) 


Immpact (Aberdeen, UK) 






Yes - review 


n/a 


Public 


Spertus et ai. (2005) 


AHA (Dallas, TX, US) 






No 


n/a 


Public 


Topic papers 


Bonow et al. (2005) 


AHA (Dallas, TX, US) 


Heart failure 


Hospital/ 

outpatient 

care 


Yes - report 


No 


Public 


Burge ef al (2007) 


CCQRT (Toronto, CA) 


Heart failure 


Primary care 


No 


No 


Public 


Campbell et ai. 
(1999) 


NPCRDC (Manchester, UK) 


CHD, Type 2 
Diabetes, Asthma 


Primary care 


Yes - original 
article 


No 


Unclear 


Desch ef al. (2008) 


RPCI (Buffalo, NY, US) 


Breast cancer 


Hospital care 


Yes - special article 


No 


Public 


Draskovic ef al. 
(2008) 


IQ healthcare (Nijmegen, NL) 


Dementia 


Hospital care 


No 


No 


Public 


Estes ef a/. (2008) 


AHA (Dallas, TX, US) 


Atrial fibrillation 


Outpatient 
care 


Yes - report 


No 


Public 


Forbes ef al. (1997) 


KU School of Nursing (Kansas City, 
MO, US) 


Stroke 


Rehabilitation 


No 


No 


Public 


Giesen ef al. (2007) 


IQ healthcare (Nijmegen, NL) 


Prescribing and 
referra 


Emergency 
primary care 


No 


No 


Unclear 


Hadorn ef o/. (1996) 


RAND (Santa Monica, CA, US) 


Heart failure 


Primary care 


Yes - article 


No 


Combined 

public/ 

private 


Hardy and Hadley 
(1995) 


CCQE (Washington, DC, US) 


Pain 


All 


No 


No 


Unclear 


Hermanides ef al. 
(2008) 


IQ healthcare (Nijmegen, NL) 


Urinary tract 
infection 


Hospital care 


Yes - major article 


No 


Unclear 


Hermens ef al. 
(2006) 


IQ healthcare (Nijmegen, NL) 


Lung cancer 


Hospital care 


Yes - article 


No 


Public 


James et ai. (1997) 


Qffice of Rural Health (Buffalo, NY, 
US) 


Heart failure 


Primary care 


Yes - paper 


No 


Public 


Kongnyuy and van 
den Broek (2008) 


LSTM (Liverpool, UK) 


Perinatal care 


Hospital care 


Yes - research 
article 


No 


Combined 

public/ 

private 


Krumholz ef al. 
(2006) 


AHA (Dallas, TX, US) 


Myocardial infarction 


Hospital care 


Yes - report 


No 


Public 
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Table 2 Characteristics of included references: General characteristics and quality assessment (Continued) 



Lee et al. (2003) 


CCORT (Toronto, CA) 


Heart failure 


Hospital/ 

outpatient 

care 


Yes - clinical study 


No 


Public 


Maclean et ai. 
(2004) 


RAND (Santa Monica, CA, US) 


Rheumatoid arthritis 


All 


Yes - original 
article 


No 


Unclear 


Martirosyan et al. 
(2008) 


iQ healthcare (Nijmegen, NL) 


Type 2 Diabetes 


Primary care 


Yes - original 
research 


No 


Public 


Mourad ef a/. 
(2007) 


iQ healthcare (Nijmegen, NL) 


Subfertility care 


All 


No 


No 


Public 


Nijkrake ef al. 
(2009) 


iQ healthcare (Nijmegen, NL) 


Parkinson's disease 


Physiotherapy 


No 


No 


Public 


Ouwens ef al. 
(2007) 


iQ healthcare (Nijmegen, NL) 


Head and neck 
cancer 


Cross-sectoral 
care 


Yes - original 
article 


No 


Public 


Ouwens ef al. 
(2010) 


iQ healthcare (Nijmegen, NL) 


Patient-centered care 


All 


Yes - original 
article 


No 


Unclear 


Radtke et a/. (2009) 


CVderm (iHamburg, DE) 


Psoriasis vulgaris 


All 


Yes - original 
paper 


No 


Unclear 


Redberg ef al. 
(2009) 


AHA (Dallas, TX, US) 


Cardiovascular 
prevention 


All 


Yes - report 


No 


Public 


Schouten ef al. 
(2005) 


iQ healthcare (Nijmegen, NL) 


Pneumonia 


Hospital care 


yes - major article 


No 


unclear 


Sugarman et o/. 
(2003) 


Qualis Heaith (Seattie, WA US) 


Dialysis 


All 


Yes - special article 


Yes 


Public 


Thomas et al. 
(2007) 


AHA (Dallas, TX, US) 


Cardiovascular 
diseases 


Rehabilitation 


No 


No 


Public 


Tu ef a/. (2008) 


CCORT (Toronto, CA) 


Myocardial infarction 


Hospital care 


Yes - review 


No 


Public 


van den Boogaard 
et al. (2010) 


iQ healthcare (Nijmegen, NL) 


Miscarriage 


All 


Yes - article 


No 


Public 


van Hulst et al. 
(2009) 


iQ healthcare (Nijmegen, NL) 


Rheumatoid arthritis 


All 


Yes - extended 
report 


No 


Unclear 


Wang et o/. (2006) 


RAND (Santa Monica, CA, US) 


Preterm birth 


Outpatient 
care 


Yes - article 


No 


Public 


Yazdany ef al. 
(2009) 


UCSF (San Francisco, CA, US) 


Lupus 

erythematodes 


All 


Yes - original 
article 


No 


Unclear 


Method + topic 
papers 


Advani ef al. (2003) 


BMiR (Stanford, CA US) 


Hypertension 


All 


No 


No 


Public 


Duffy ef al. (2005) 


APiRE (Arlington, VA US) 


Bipolar disorder 


Outpatient 
care 


No 


No 


Unclear 


Golden et o/. (2008) 


UAMS (Littie Rock, US) 


Bipolar disorder 


Outpatient 
care 


No 


No 


Public 


Hutchinson ef al. 
(2003) 


ScHARR (Sheffield, UK) 


CHD 


Primary care 


Yes - original 
paper 


Yes 


Combined 

public/ 

private 


LaClair ef al. (2001) 


VA Medical Center (Kansas City, MO, 
US) 


Stroke 


Rehabilitation 


No 


No 


Public 


Wollersheim et o/. 
(2007) 


IQ healthcare (Nijmegen, NL) 


Oncology, Type 2 
Diabetes, pneumonia 


All 


Yes - review article 


No 


Unclear 



AZQ = Arztliches Zentrum fiir Qualitat in der Medizin (Agency for Quality in Medicine); AHCPR = Agency for Healthcare Policy and Research; AHRQ = Agency for 
Healthcare Research and Quality; AQUA-lnstitute = Institute for Applied Improvement and Research in Health Care; DCRI = Duke Clinical Research Institute; 
NPCRDC = National Primary Care Research and Development Council; Immpact - Initiative for Maternal Mortality Programme Assessment; CCORT = Canadian 
Cardiovascular Outcomes Research Team; CHD = coronary heart disease; RPCI = Roswell Park Cancer Institute; AHA = American Heart Association; CCQE = Center 
for Clinical Quality Evaluation; LSTM = Liverpool School of Tropical Medicine; CVderm = Competenzzentrum Versorgungsforschung in der Dermatologle (Institute 
for Health Services Research in Dermatology}; UCSF = University of California, San Francisco; BMIR = Center for Biomedical Informatics Research; APIRE = 
American Psychiatric Institute for Research and Education; UAMS = University of Arkansas for Medical Sciences; ScHARR = School of Health and Related Research. 



Table 3 Characteristics of included references: Methodological approaches to topic/guideline selection and extraction of recommendations 





Topic/guideline selection 










Extraction of recommendations 




Reference 


Criteria for 
selection of 
topic 


Development 
of Ql from... 


Criteria for 
selection of 
participants 


Criteria for 
selection of 
guidelines 


Participants 
listed" 


Critical appraisal 


Guidelines Extraction of all/ 
listed" a selection of 

recommendations 


Criteria for 

recommendation 

selection'' 


Potential 
indicators 
listed" 


Method 
papers 


AZQ (2009) 


No 


One guideline 


No 


No 




No 


Unclear 






AHCPR 
(1995) 


No 


One guideline 


Yes 

Profession 
involved in the 


Yes 

Methodological 
quality 




Yes 

Not detailed 


Selection 


Yes 

Impact on patient 
outcome 





selected 

healthcare 

process, 

methodological 
competence 

AHRQ(1995) Yes More than No 

Regulatory one guideline 

requirements, 
quality gap, 
guideline 
adherence 
unknown 



Yes - Yes - Selection Yes 

Methodological Not detailed Impact on patient 

quality outcome and relevance 

to obtaining value for 
money 



AQUA (2010) 


Yes 

Public health 
relevance, sound 
evidence base, 
feasibility 


Guidelines 
and other 
sources 


No 


Yes 

Methodological 
quality 


Yes 

AGREE Instrument 


All 


Baker and 
Fraser (1 995) 


No 


Not specified 

(method 

paper) 


No 


No 


Yes 

Not detailed 


Unclear 


Bergman 
(1999) 


Yes 

Sound evidence 
base 


Not specified 

(method 

paper) 


No 


No 


Yes 

Not detailed 


Unclear 



Califfefo/. No One guideline No No - Yes - Selection Yes 

(2002) Not detailed Level of evidence 



Campbell et No Not specified No No - No - Unclear 

al. (2002) (method 

paper) 

Graham et Yes Guidelines No No - No - Unclear 

al. (2009) Ouality gap and other 

sources 



Spertus et al. No 
(2005) 



Not specified 

(method 

paper) 



No 



Yes 

Strength of 

evidence, 
clinical 
relevance, 
magnitude of 
relationship 
between 
performance 
and outcome 



Yes 

Not detailed 



Selection 



Yes 

Level of evidence, 
impact on patient 
outcome 



Table 3 Characteristics of included references: Methodological approaches to topic/guideline selection and extraction of recommendations (Continued) 



Topic 
papers 

Bonow ef al. Yes 

(2005) Public health 

relevance, 
quality gap, 
costs 



More than 
one guideline 



No 



No 



Yes 



Yes 

Not detailed 



Yes 



Selection 



Yes 

Grade of 
recommendation, 
relevance for the topic 



No 



Burge et a/. Yes 

(2007) Public health 

relevance, 
quality gap 



Unclear 



No 



No 



Yes 



No 



No 



Selection 



Yes 

Potential for 
improvement, 
meaningful, valid, 
reliable, adjustable, 
feasible 



No 



Campbell et 
al. (1999) 



Yes 

Public health 
relevance, 
substantial 
amount of 
workload in 
general practice 



Guidelines 
and other 
sources 



No 



No 



No 



No 



Yes 



Unclear 



No 



Desch ef al. No 


Guidelines 


Yes No 


Unclear 


No 


Yes 


Selection 


Yes No 


(2008) 


and other 


Profession 










Impact on patient 




sources 


involved in the 










outcome, potential for 






selected 










improvement, feasibility 






healthcare 










of data collection 






process 













Draskovic et 
al. (2008) 



Yes 

Variance in 
quality of care 
between 
providers 



One guideline No 



No 



No 



No 



Yes 



Unclear 



No 



Estes ef al. 
(2008) 



Yes 

Public health 
relevance and 
costs 



Guidelines 
and other 
sources 



No 



No 



Yes 



Yes 

Not detailed 



Yes 



Selection 



Yes 

Grade of 
recommendation, 
relevance for the topic 



No 



Forbes ef al. 
(1 997) 



Yes 

Public health 
relevance, 
individual 
impact on 
quality of life 



One guideline No 



No 



No 



No 



Yes 



All 



No 



Giesen et al. 
(2007) 



Yes 

Quality of care 
unknown 



Guidelines 
and other 
sources 



No 



Yes 

Applicability to 
the setting, 
clinical 
relevance 



Yes 



Yes 

AGREE instrument 



Yes 



Selection 



Yes 

Relevance for the 
selected topic 



No 



Hadorn ef al. 
(1 996) 



Yes 

Public health 
relevance, 
individual 
quality-of-life 
impact, costs 



One guideline No 



No 



Yes 



No 



Yes 



All 



Yes 
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Hardy and 

HaHlpv 

(1995) 


No 


One guideline 


No 


Unclear 


No 


No 


Yes 


Unclear 




No 


Hermanides 
ef al. (2008) 


Yes 

Public health 
relevance, 
quality gap 


One guideline 


No 


No 


No 


No 


Yes 


Selection 


No 


Yes 


Hermens ef 


Yes 


One guideline 


No 


No 


No 


No 


Yes 


All 




No 



al. (2005) Ouality of care 
unknown, 
guideline 
adherence 
unclear 



James ef al. Yes 

(1997) Public health 

relevance, costs, 
quality gap 



One guideline No 



No 



No 



Yes 

Not detailed 



Yes 



All 



No 



Kongnyuy 
and van den 
Broek (2008) 



No 



Guidelines 
and other 
sources 



No 



No 



No 



No 



Yes 



Unclear 



No 



Krumholz ef Yes 

al. (2006) Public health 
relevance, 
quality gap 



More than 
one guideline 



No 



No 



Yes 



Yes 

Not detailed 



Yes 



Selection 



Yes 

Grade of 
recommendation 



No 



Lee ef al. 
(2003) 



No 



Guidelines 
and other 
sources 



No 



No 



No 



No 



Yes 



Unclear 



No 



Maclean ef 
al. (2004) 



Yes 

Public health 
relevance 



Guidelines 
and other 
sources 



No 



No 



No 



Unclear 



Yes 



Selection 



Yes 

Impact on patient 
outcome, grade of 
recommendation 



No 



Martirosyan Yes More than No No No No Yes Selection Yes Yes 

ef al. (2008) Public health one guideline Measurability 

relevance, 

quality of care 

unknown 



Mourad ef 
al. (2007) 



Yes 

Public health 
relevance, 
quality gap 



More than 
one guideline 



No 



Yes No 

Methodological 

quality 



No 



Yes All 



No 



Nijkrake ef 
al. (2009) 



Yes 

Public health 
relevance and 
complexity of 
the topic 



One guideline No 



No 



No 



No 



Yes Selection 



Yes 

Acceptability, 
measurability 



No 



Ouwens ef 
al. (2007) 



Yes 

Complexity of 
the process of 
care 



Guidelines 
and other 
sources 



No 



No 



No 



No 



Yes Selection 



Yes 

Impact on patient 
outcome 



No 
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Ouwens ef 


Yes 


Guidelines 


No 


Yes 


No 


No 


Yes 


All 


- 


No 


al. (2010) 


hdividua 


and other 




Applicability to 
















impact on 


sources 




the setting 
















LjUdllLy Ul lllfcr, 






















LjUdllLy ydfJ 




















Radtke et al. 


No 


Guidelines 


No 


No 


No 


Yes 


Yes 


Unclear 




No 






and other 








iNot Getaiieo 














sources 


















Red berg ef 


Yes 


Qne guideline 


No 


No 


1 ,0 


No 


Yes 


Selection 


Unclear 


No 


01. ^zuuyj 


rUDiic neaitn 






















relevBDce, costs, 






















quality gap 




















Schouten et 


Yes 


Guidelines 


No 


No 


No 


No 


Yes 


Selection 


No 


Yes 


al. (2005) 


Quality gap 


and other 






















sources 


















Sugarman ef 


Yes 


Qne guideline 


No 


No 


No 


No 


Yes 


Unclear 


- 


No 


al. (2003) 


Quality of care 






















unknown, 






















regulatory 






















requirements 




















Thomas ef 


Yes 


Guidelines 


No 


No 


Yes 


Yes 


Yes 


Selection 


Yes 


No 


al. (2007) 


Underutilization, 


and other 








Not detailed 






Grade of 






quality of care 


sources 














recommendation, level 






unl<nown 
















of evidence 




Tu ef al. 


Yes 


Guidelines 


No 


No 


Yes 


No 


Yes 


Selection 


Yes 


No 


(2008) 


Quality gap 


and other 














Meaningful, valid and 








sources 














reliable, feasible. 






















accountable for patient 






















variability, potential for 






















improvement. 




van den 


Yes 


Qne guideline 


No 


Yes 


No 


No 


Yes 


All 


- 


No 


Boogaard ef 


Quality gap 






Most recently 














al. [zu 1 uj 








revised 






















n 1 MM i nj^ 
y u lUCI 1 1 IC 






















aval able 














van Hulst ef 


No 


Guidelines 


No 


No 


No 


No 


Yes 


Selection 


Yes 


No 


al. (2009) 




and other 














Grade of 








sources 














recommendations 




Wang ef al. 


Yes 


Guidelines 


No 


No 


Yes 


No 


No 


Selection 


Yes 


No 


(2005) 


Public health 


and other 














Impact on patient 






relevance, 


sources 














outcome, level of 






complex process 
















evidence, potential for 






of care, quality 
















improvement, feasibility 






gap 
















of data collection 




Yazdany ef 


Yes 


Guidelines 


No 


Yes 


Yes 


Unclear 


No 


Selection 


Yes 


No 


al. (2009) 


Quality of care 


and other 




Methodological 










Eligible population, 






unknown 


sources 




quality 










process of care 





performed by 
healthcare providers, 
impact on patient 
outcome 
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Method + 
















topic 
















papers 
















Advani et ai. No 


One guideline No 


No 


No 


No 


Yes 


Unclear 


No 


(2003) 

















Duffy et o/. 
(2005) 



Yes 

hdividua 
impact on 
quality of life, 
quality gap 



More than No 
one guideline 



No 



No 



No 



Yes Selection Yes 

Level of evidence, 
impact on patient 
outcome, breadth of 
available treatment 
recommendations, 
clinical utility and 
appropriateness, 
proportion of patients 
for whom the 
recommendation is 
likely to be relevant 



No 



Golden et al. 


Yes 


Guidelines 


Yes 


No 


No 


No No 


Selection 


Yes No 


(2008) 


Public health 


and other 


Profession 










Level of evidence 




relevance, costs, 


sources 


involved in the 














quality gap 




selected health 


















care process 












Hutchinson 


No 


More than 


No 


Yes 


No 


Yes Yes 


Selection 


Unclear No 


et al. (2003) 




one guideline 




Evidence based 




Suitable for primary 







care, agency responsible 
for development clearly 
identifiable, objectives 
clearly defined, 
independent review 
prior to publication, 
information regarding 
evidence adequate and 
explicit, link between 
major recommendations 
and underlying 
evidence 



Laclair et al. No 
(2001) 



One guideline No 



No 



Yes 



No 



Yes 



All 



No 



Wollersheim Yes Guidelines 
et al. (2007) Ouality gap, and other 

public health sources 

relevance, sound 

evidence base 



Yes 

IVlembership in a 
guideline- 
development 
committee, 
methodological 
competence, 
profession 
involved in the 
selected 
healthcare 
process 



No 



No 



No 



Yes 



Unclear 



No 



Ql = quality indicator; AZQ = Arztliches Zentrum fiir Qualitat in der Medizin (Agency for Quality in Medicine); AHCPR = Agency for Healthcare Policy and Research; AHRQ = 
Quality; AQUA-lnstitute = Institute for Applied Improvement and Research in Health Care; AGREE = Appraisal of Guidelines for Research and Evaluation in Europe. 
^Does not apply to method papers; '^does apply if not all recommendations are extracted. 



Agency for Healthcare Research and 
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Ql selection Additional Ql development elements 



Reference 


Panel 
method 


Criteria for panel 
members 


Panel 

members 

listed" 


Selected 

indicators 

listed" 


Sources 
transparent' 


LoE" 


Rating criteria 


Practice 
test 


Implementation strategy 


Patient 
participation 


Method 
papers 


AZQ (2009) 


Unclear 


Unclear 








Yes 


Yes 

Importance for the healthcare system, 
clarity, improvability, risk for adverse 

diCL-L, cviUdiL.c: LJajC, yiauc ui 

recommendation 


Proposed 


No 


No 


AHCPR 
(1995) 


No 


WU [Jdlltrl IIItrLI lUU 








No 




Not 

mentioned 


No 


Nn 


AHRO f1 QQ5^ 


No 


\\\r\ nanf^ mpthnH 

1 \U [JOl Id 1 1 ICLI lUU 








No 


No 


I nr" 1 iHpH 

I I IL,I UUd_l 


Yes 

Development of data collection 
software, audit and feedback 


No 


AQUA (2010) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise, 

methodologica 

expertise 


- 


- 


- 


Yes 


Yes 

Relevance, clarity, feasibility 


ncluded 


Yes 

Development/upgrading of 
data collection software 


Ql selection 


Baker and 
Fraser (1995) 


No 


No panel method 


- 


- 


- 


No 


Unclear 


Not 

mentioned 


Yes 

Local development, ownership 


No 


Rcirn ma n 
uci y 1 1 Id 1 1 

(1999) 


No 


l\U IJOl Id 1 1 ICLI lUU 








Yes 




r 1 ukju jcu 


Yes 

nvolving key stakeholders 


No 


Califf ef a/. 
(2002) 


No 


No panel method 








Yes 


Unclear 


Not 

mentioned 


Yes 

Education and feedback 


No 


Campbell ef 
al. (2002) 


Other 


Unclear 








No 


Unclear 


Not 

mentioned 


No 


No 


Graham ef 
al. (2009) 


Other 


No 








No 


Yes 

Grade of recommendation, level of 
evidence, measurability, improvability 


Included 


Yes 

Audit and feedback 


No 


Spertus ef al. 
(2005) 


No 


No panel method 








No 


Yes 

Useful in improving patient outcomes, 
measure design, measure 
implementation, overall assessment 


Not 

mentioned 


No 


No 


Topic 
papers 


Bonow ef al. 
(2005) 


Other 


No 


Yes 


Yes 


Yes 


Yes 


Yes 

Useful in improving patient outcomes, 
measure design, measure 
implementation, overall assessment 


Not 

mentioned 


Yes 

Defining challenges to 
implementation for each Ql 


No 


Burge ef al. 
(2007) 


Modified 

RAND/ 

UCLA 


Yes 

Members of 
specialist societies 


Yes 


Yes 


In part 


No 


No 


Proposed 


No 


No 


Campbell ef 
al. (1999) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise, 
members of 
specialist societies 


No 


Yes 


n part 


Yes 


No 


Not 

mentioned 


Yes 


No 
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Desch et ai. 
(2008) 


Otiier 


Yes 

Members of 

specialist 

societies, 

methodologica 

expertise 


Yes 


Yes 


Yes 


No 


No 


Not 

mentioned 


Yes 

Integration in nationwide 
quality-improvement programs 


No 


Draskovic et 
al. (2008) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise 


No 


Yes 


Yes 


No 


Yes 

Face validity 


ncluded 


Yes 

Including the informal 
caregivers' perspective 


No 


Estes ef al. 
(2008) 


Otiier 


No 


Yes 


Yes 


Yes 


Yes 


Yes 

Useful to improve patient outcomes, 
measure design, measure 
implementation, overall assessment 


Not 

mentioned 


Yes 

Defining challenges to 
implementation for each Ql 


No 


Forbes ef al. 
(1997) 


No 


No panel method 


No panel 
method 


Yes 


Yes 


No 


No 


ncluded 


Yes 

Pilot testing 


No 


Giesen et ai. 
(2007) 


Otiier 


Unclear 


No 


Yes 


n part 


No 


Yes 

Relevance, utility for evaluation of care 


ncluded 


No 


No 


Hadorn et al. 
(1996) 


Unclear 


No 


!o 




n part 


No 


Unclear 


Not 

mentioned 


No 


lo 


Hardy and 

Hadley 

(1995) 


Unclear 


Unclear 


No 


No 


Yes 


No 


No 


Not 

mentioned 


No 


No 


Hermanides 

ef al. (2008) 


Other 


Yes 

Clinical expertise 


Yes 


Yes 


Yes 


Yes 


Yes 

Appropriateness 


ncluded 


No 


No 


Hermens ef 
a/. (2006) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise 


Yes 


Yes 


Yes 


No 


Yes 

Professional quality, organisational 
quality, patient-oriented quality 


Included 


Yes 

Practice test 


Ql selection 


James ef al. 
(1997) 


Other 


Yes 

Clinical expertise 


No 


Yes 


Yes 


Yes 


Yes 

Educational appropriateness, clinical 
importance, measurement feasibility 


Not 

mentioned 


No 


No 


Kongnyuy 
and van den 
Broek (2008) 


Other 


Yes 

Clinical expertise, 
laypersons 


No 


Yes 


h part 


No 


No 


Planned 


Yes 

nvolving all grades of health 
professionals during the whole 
development process 


Ql selection 



Krumholz et Other Yes Yes 
al. (2006) Clinical expertise, 

methodologica 
expertise 
members of 
specialist societies 



Yes Yes Yes Yes 

Useful in improving patient outcomes, 
measure design, measure 
implementation, overall assessment 



Not Yes 

mentioned Defining challenges to 

implementation for each Ql 



No 



3- 7^ 

i £ 
II 

51 3" 



i 2 



Lee ef al. 
(2003) 



Other 



Yes Yes 
Clinical expertise 



Yes In part No Yes Not No 

Meaningfulness, usefulness, potential for mentioned 
improvement, impact on patient 
outcomes, feasibility of data collection 



No 



o 
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Maclean et 


Modified 


Yes 


Yes 


Yes 


No 


Yes 


Unclear 


Not 


No No 


al. (2004) 


RAND/ 


linical expertise, 












mentioned 






UCLA 


methodological 




















expertise 




















members of 




















specialist societies 
















Martirosyan 


Modified 


Yes 


No 


Yes 


In part 


No 


Unclear 


Included 


No No 


et al. (2008) 


RAND/ 


Clinical expertise, 


















UCLA 


methodological 




















expertise 




















members of 




















specialist societies 
















Mourad et 


Modified 


Yes 


No 


Yes 


Yes 


Yes 


Unclear 


Proposed 


Yes No 


a!. (2007) 


RAND/ 


Clinical expertise, 














Practice test 




UCLA 


methodo ogical 




















expertise 
















Nijkrake et 


Other 


Yes 


No 


No 


Yes 


Yes 


Yes 


Included 


Yes No 


al. (2009) 




Clinical expertise, 










Relevance (effectiveness, efficiency, 




Training in the correct use of 






rtiethodo ogical 










acceptability, measurability) 




the respective guide ine 






t:A[Jt:i Liic 
















Ouwens et 


Modified 


Yes 


No 


Yes 


In part 


No 


Yes 


Included 


Yes Ql selection 


01. [ZUU/) 


DA Mn/ 
nANU/ 


Clinica expertise 










Clinically relevant to patients' hea th 




Practice test 




1 1^1 A 












benefits and/or to the continuity and 




















coordination of care 






Ouwens et 


Other 


Yes 


No 


Yes 


In part 


No 


Unclear 


Included 


Yes Ql selection 


al. (2010) 




Patient 














Patient participation 






representatives 
















Radtke et al. 


Other 


Yes 


No 


Yes 


In part 


No 


Yes 


Included 


No No 


(2009) 




Clinical expertise. 










nclusion in the research literature, 










methodological 










measurable under routine conditions. 










expertise, patients 










inclusion in a certain high-quality 




















guideline, reproducibility, validity, clinical 




















relevance, sensitivity to change 






Red berg et 


Other 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Not 


No No 


al. (2009) 




Clinical expertise. 










Useful in improving patient outcomes, 


mentioned 








methodological 










measure design, measure 










expertise 










implementation, overall assessment 










membership in 




















specialist societies 
















Schouten et 


Modified 


Yes 


No 


Yes 


Yes 


Yes 


Yes 


Included 


No No 


al. (2005) 


RAND/ 


Clinical expertise. 










Clinical relevance to the patient's health 








UCLA 


methodological 










benefit, relevance to reducing 







expertise antimicrobial resistance, relevance to 

cost effectiveness 
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Sugarman ef 


Other 


Yes 


No 


No 


Yes 


Yes 


Yes 


Included 


No 


No 


al. (2003) 




Clinical expertise, 










Clinical importance, feasibility of 












membership in 










measurement, level of evidence 












specialist societies 


















Thomas ef 


Unclear 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Not 


Yes 


No 


al. (2007) 




Clinical expertise, 










Evidence based, interpretable, actionable, 


mentioned 


Defining challenges to 








methodological 










clinically meaningful, valid, reliable, 




implementation for each Ql 








expertise. 










feasible 












membership in 






















specialist societies 


















Tu ef al. 


Other 


Yes 


Yes 


Yes 


In part 


No 


Yes 


Not 


Yes 


No 


(2008) 




Clinical expertise. 










Usefulness in improving patient 


mentioned 


Pay for performance. 








methodological 










outcomes, feasibility of data collection. 




collaboration with national and 








expertise. 










reliabi ity, validity 




local initiatives, use of standard 








membership in 














tools, presentation at scientific 








specialist societies 














meetings, aval ability online 




van den 


Modified 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Proposed 


No 


No 


Boogaard ef 


KAN U/ 


Clinical expertise 










Health gain, overall efficacy 








ai. 1 


1 in A 




















van Hulst ef 


Modified 


Yes 


No 


Yes 


In part 


Yes 


No 


Not 


Yes 


No 


al. (zooyj 


D A M P\ / 

KANU/ 


Clinical expertise. 












mentioned 


Using understandable and 






UCLA 


methodo ogical 














measurable Qls 








expertise 


















Wang ef al. 


Other 


Yes 


No 


Yes 


In part 


Yes 


Yes 


Not 


No 


No 






Membership in 










Validity, feasibility 


mentioned 










cnpri?ilit;t 'soriptip'^ 


















Yazdany ef 


Modified 


Yes 


Yes 


Yes 


No 


Yes 


Yes 


Proposed 


Yes 


No 


al. (2009) 


RAND/ 


Clinical expertise. 










Evidence base, validity, feasibility 




Assess the technical 






UCLA 


methodological 














characteristics of developed Qls 








expertise 


















Method + 






















topic 






















papers 























Advani ef al. No No panel method No panel No Yes No No Included No 

(2003) method 



Duffy ef al. 
(2005) 



Unclear Unclear 



No 



Yes 



Yes 



Yes 



Unclear 



Planned Yes 

Integration in health plan 
performance measurement, 
quality monitoring and 
accreditation programs, 
integration of needed data 
elements in medical 
information systems 



No 



Table 4 Characteristics of included references: Methodological approaches to Ql selection, practice test, and implementation (Continued) 



Golden ef al. 
(2008) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise, 
methodological 
expertise, 
laypersons 


No 


No 


In part 


No 


Yes 

Meaningfulness, quality gap, 
improvability, feasibility of data collection 


Included 


Yes 

Transparency during the 
development process, 
providing the data collection 
tool, submission to a national 
performance measurement 
program 


Ql selection 


Hutchinson 
ef al. (2003) 


Other 


Yes 

Clinical expertise 


No 


Yes 


In part 


Yes 


No 


Not 

mentioned 


No 


No 


Laclair et al. 
(2001) 


Other 


Yes 

Clinical expertise, 

methodological 

expertise 


No 


No 


Yes 


Yes 


No 


Included 


No 


No 


Wollersheim 
ef al. (2007) 


Modified 

RAND/ 

UCLA 


Yes 

Clinical expertise, 

methodological 

expertise 


No 


Yes 


In part 


Unclear 


No 


Included 


Yes 

Periodic audits 


No 



Ql = quality indicator; AZQ = Arztliches Zentrum fur Qualitat in der Medizin {Agency for Quality in Medicine); AHCPR = Agency for Healthcare Policy and Research; AHRQ = Agency for Healthcare Research and 
Quality; AQUA-lnstitute = Institute for Applied Innprovement and Research in Health Care. 

^Does not apply to method papers; "^LoE = Level of evidence {reported for underlying recommendations of the Ql). 
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♦ member of a guideline development committee, 

♦ having methodological competence, 

♦ belonging to a profession involved in the selected 
healthcare process. 

Extraction of recommendations 

Nine studies extracted all recommendations from 
selected guidelines. In 25 studies, recommendations 
were selected during the extraction process and not all 
recommendations were extracted as potential QIs. Cri- 
teria for this selection were reported in 21 of the 25 stu- 
dies. Criteria for the preselection at the stage of 
recommendation extraction mentioned by the Agency 
for Healthcare Research and Quality (AHRQ) are 

♦ the size of the impact on patient health (the AHRQ 
considers the impact great when an issue affects a few 
patients severely or affects many patients), 

♦ the relevance to obtaining value for money. 

Other criteria for the preselection formulated by 
Hadorn et al. [39] are 

♦ the importance to quality of healthcare provided, 

♦ the feasibility of monitoring. 

Other frequently reported criteria were the level of 
evidence, the grade of recommendation, and 
measurability. 

Levels of evidence and grades of recommendation of 
the recommendations potential QIs were developed 



from were reported in 24 studies. Only four studies 
reported criteria for the selection of persons who 
extracted potential QIs from guidelines. They were simi- 
lar to those for persons involved in guideline selection 
(see above); both tasks were usually carried out by the 
same group of people. 

The AHRQ [24] provides a detailed description of the 
extraction process, including specifications of partici- 
pants' necessary skills, as well as criteria for the selec- 
tion of recommendations to be extracted. 

Four requirements for persons involved in the extrac- 
tion of potential QIs from guidelines postulated by the 
AHRQ are 

• clinician and nonclinician management skills, 

• clinical expertise, 

• technical expertise in performance measurement, 

• healthcare information management expertise. 
Another prerequisite for a valid extraction process 

mentioned in several of the relevant studies requires 
that the extraction be performed by at least two 
researchers independently [25,37-39]. 

Ql selection 

In 35 studies, a consensus method was used to augment 
the evidence from literature with expert and layperson 
opinion by letting a panel rate and select a set of final 
QIs from a set of potential QIs. In 15 of these 35 













Extraction of 


QI selection 




recommendations 








•Panel (selected following 




•Panel (selected 


certain criteria) 




following certain 


•Clinical expertise 




criteria) 


•Membership in a 




•Clinician and 


specialist society 




nonclinician 


•Methodological 




management skills 


expertise 




•Clinical expertise 


•With or without patient 




•Expertise in 


participation 




performance 
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Figure 3 Methodological variability of guideline-based Ql development. 
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publications this method was described as the "modified 
RAND/UCLA method," named after the RAND/UCLA 
(University of Cahfornia, Los Angeles) appropriateness 
method [68]. 

Whereas only a few studies named the individual 
members of the panels, criteria for their selection {e.g., 
clinical expertise, methodological expertise, membership 
in a specialist society) were reported in 31 of 35 studies. 
Only 25 of 35 studies provided rating criteria for the 
panel process. Among the frequently named criteria 
were the usefulness of QIs for improving patient out- 
comes, their relevance, and the feasibility of monitoring. 

Participation of patients in the development process 
was reported in six studies. In all of these studies, 
patients participated in the panels. No study reported 
patient participation during guideline selection and the 
extraction of recommendations. 

Practice test 

Only 19 studies reported the conduct of a QI practice 
test. In two studies, the practice test was conducted 
after the development process was completed. In 21 stu- 
dies, a practice test was not mentioned at all. 

Implementation 

An implementation strategy for guideline-based QIs was 
reported in 26 studies. Among the reported activities 
were the instruction of key persons ("early adopters") as 
multipliers, the participation of end users in the devel- 
opment process, the publication of developed QIs by 
medical associations, supplying the appropriate software, 
and the adaptation of "global" QIs to more specific set- 
tings. Financial incentives and certification were also 
used to support implementation. 

Discussion 

Topic selection 

Authors tended to describe the process of topic selec- 
tion in insufficient detail. Mostly, selection criteria 
merely reflected the aims of the application of QIs in 
general: to measure and improve quality in areas of 
healthcare where the actual quality of care is either sub- 
optimal or unknown. 

Guideline selection 

The selected literature describes two different 
approaches to guideline selection. The first approach 
identified in the reviewed literature is to develop QIs 
based on one or only a few preselected guidelines, often 
with the aim of supporting or evaluating guideline 
implementation. In certain contexts, such as specific set- 
tings in small healthcare systems, only one guideline 
may be available for QI development. In these cases, 
guideline-selection processes are of no or only minor 



relevance, and the number of recommendations to be 
translated into potential QIs is proportionately low. 

The second approach is to select a clinical topic and, 
subsequently, to obtain suitable, topic-specific guidelines 
as a basis for the development of QIs from guideline 
recommendations. In this case, expert opinion and exist- 
ing QI sets are sometimes used as alternative sources for 
QIs. In comparison to the first approach, this approach 
provides a broader basis for the subsequent develop- 
ment of QIs, bears the potential to produce a balanced 
set of QIs, carries a reduced risk of selection bias, and 
increases content validity. 

Many studies do not describe their guideline- selection 
criteria in sufficient detail and lack critical appraisal of 
their selected guidelines, both of which may compro- 
mise content validity and hence the quality of resulting 
QI sets. We argue that high-quality QIs can only be 
derived from high-quality guidelines. To ensure QIs ori- 
ginate from a sound foundation, development commit- 
tees should (a) conduct a systematic search for relevant 
guidelines in national and international guideline data- 
bases as well as conventional literature databases and 
(b) conduct a critical appraisal of the methodological 
quality of selected guidelines {e.g., by using the AGREE 
instrument) [67]. 

As is common practice in other areas of research such 
as guideline development, the documentation of selec- 
tion criteria for participating persons as well as the dis- 
closure of their names and potential conflicts of interest 
could greatly add transparency to the whole develop- 
ment process and, as a result, increase the content valid- 
ity of QIs. 

Extraction of recommendations 

The main focus of this review is the extraction of guide- 
line recommendations. This step is both crucial and 
unique to guideline-based QI development, whereas the 
other steps could also be applied to the development of 
QIs from other sources such as primary literature or 
existing QI sets. We only included studies that provided 
a detailed description of the recommendation-extraction 
process. As a result, we excluded a large number of 
otherwise eligible studies (see Additional file 4: Table 4 
for a list of studies excluded for this reason). 

The reviewed literature describes two different 
approaches to the extraction of guideline recommenda- 
tions. The first approach is to initially extract all recom- 
mendations and to then select QIs using a systematic 
consensus process. The second approach is to select a 
limited number of recommendations during the extrac- 
tion process. We believe the difference between both 
approaches is of crucial importance to the quality of 
ensuing QI sets. Predominantly, only a small number of 
persons conduct the extraction process. Often, those 
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participants were not selected following transparent 
selection criteria. The extraction of potential QIs itself 
through this small group of participants usually does 
not follow any documented selection criteria, either. As 
a result, the final QI set may suffer from selection bias. 

Subsequent systematic consensus processes to rate 
and select the extracted potential QIs are usually con- 
ducted by larger panels. In comparison to the small 
group of persons conducting the selection of potential 
QIs, panel participants are commonly selected to build a 
balanced panel of different professionals participating in 
the process of healthcare the QIs are developed for. In 
addition, the use of predesigned forms containing rating 
and selection criteria during these systematic consensus 
processes substantially reduces the risk of selection bias 
(see "QI selection"). 

Another important aspect of the extraction process is 
the translation of the guideline text into recommenda- 
tions manageable as potential QIs. It can be difficult to 
derive appropriate numerators and denominators on the 
basis of the guideline recommendation wording, which 
may not be specific enough for this purpose. A whole 
paragraph of guideline text, for instance, cannot easily 
be translated into a potential QI without cutting out 
potentially relevant information. Thus, the translation 
process is a further potential source of bias. 

Hence, both the selection of participants as such and 
the documentation of selection criteria for participants 
are of great importance. We identified a large deficit in 
the existing literature regarding this: Only five studies 
reported selection criteria for participants. 

QI selection 

Panel methods are not specific to guideline-based QI 
development and are frequently used to systematically 
augment the evidence from guidelines with expert opi- 
nion {e.g., the widely used RAND/UCLA appropriateness 
method [68,69]). Performed carefully, this reduces the 
risk of unintentional influence of stakeholders on the 
results of the development process [70]. Panel methods 
are an established component of the development pro- 
cess of high-quality guidelines. As our results confirm, 
they are also widely used in the development of QIs 
[65]. Many of the reviewed studies showed a lack of 
transparency regarding the nomination process {e.g., in 
not providing explicit selection criteria for panel 
members). 

Our results show that patient participation during QI 
development is extremely uncommon. In principle, the 
frequently used panel method offers room for the parti- 
cipation of patients or patient representatives. However, 
to date, no standardized approach to patient participa- 
tion during QI development exists. To fill this gap, our 
working group is currently conducting a systematic 



review of approaches to patient participation during QI 
development. 

Practice test 

Practice tests prior to publication and usage of QIs are 
an essential step in evaluating validity, reliability, feasi- 
bility, and other important attributes of QIs (see Back- 
ground). They are an integral part of any 
implementation strategy and an essential component of 
the quality loop [7,26]. The practice test in a study by 
Wollersheim et al. [10] showed that between 10% and 
20% of the developed QIs were not measurable. 

It could be argued that regular evaluations of the 
usage of QIs suffice. However, given the impact QIs can 
have from day one of their application {e.g., if used in 
pay-for-performance models [see Background]) and the 
fact that QIs are more widely accepted after an advance 
test, it is desirable that practice tests under "laboratory 
conditions" become established components of the 
development process. 

Implementation 

The importance of implementation strategies is often 
referred to in the course of critical appraisal of guide- 
lines [42]. As for guideline development, implementation 
strategies are indispensable for the real-life application 
of QIs [58]. Our results show that even though a wide 
variety of implementation strategies are reported, they 
are not always part of the QI development process. 
Given the importance of implementation, a thorough 
discussion and application of implementation strategies 
should be an integral part of a gold-standard QI devel- 
opment method. 

Strengths and limitations 

To our knowledge, this is the first systematic review of 
methodological approaches to guideline-based QI devel- 
opment. This systematic review has been conducted fol- 
lowing a rigorous methodological approach [71]. The 
identification of methodological approaches to each step 
of guideline-based QI development allows a detailed 
description and comparison of the development meth- 
ods published so far. We summarized the available evi- 
dence from systematically retrieved literature to provide 
a comprehensive overview of guideline-based QI 
development. 

A major limitation of this study is that we were not 
able to provide answers to review questions 2 and 3. 
The selected studies were very heterogeneous in type, in 
terms of the quality of reporting and in the methodolo- 
gical approaches to guideline-based QI development 
presented. Because we could not identify any studies 
comparing different methodological approaches to 
guideline-based QI development and no gold standard 
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exists to compare the published methodological 
approaches to, we were not able to provide an evidence- 
based judgment on the methodological approaches iden- 
tified. Hence, we were not able to determine whether 
any of the methodological approaches (as a whole or as 
single development steps) is "superior" to the others in 
its ability to generate high-quality QIs. 

However, in describing the methodological approaches 
used by the different working groups developing QIs, we 
provide a basis for further research. This research 
should seek to determine which of these methodological 
approaches applied to individual steps of the develop- 
ment process are best suited to constitute a develop- 
ment pathway that generates the "best" QIs. In order to 
achieve this aim in view of limited resources, existing 
guideline developers network infrastructure {e.g., the G- 
I-N) should be used to cooperate and formulate a gold 
standard, as proposed by Blozik et al. [20] . 

Conclusions 

A wide variety of methodological approaches are 
described in the literature for guideline-based QI devel- 
opment. It remains unclear which method leads to the 
best QIs, since no randomized controlled or other com- 
parative studies investigating this issue exist. 

In presenting a comprehensive methodological over- 
view, we provide a groundwork for further research 
leading to an evidence-based gold standard for guide- 
line-based QI development. 
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